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ABSTRACT 



The purpose of this paper is to describe the techniques used 
in establishing the concordance tables between the Test of English as a 
Foreign Language (TOEFL) , paper and pencil (P&P) , and computer-based testing 
( CBT) sections and total reported score scales. Listening, reading, and 
composite structure and essay scores plus a total score are reported in the 
CBT on scales that are nonoverlapping with the P&P scales. The concordance 
study was conducted to make it possible to equate the CBT test to the P&P 
test. Examinees (n=8,387) took the CBT soon after their P&P tests. Responses 
from both were used to estimate the conditional distributions of the CBT 
section scores and the observed Essay scores, given the P&P scores. Projected 
population distributions were then obtained for the CBT scores and observed 
Essay scores. Given reference forms for the CBT, the CBT and Essay score 
distributions were projected to reference- test observed score distributions 
that then served as the basis for concordance functions. Concordance tables 
were successfully established between the CBT and the P&P reported score 
scales for TOEFL. It might be argued that the researchers could have used the 
study group alone to establish the concordance relationships. However, 
because of the self -selection of the examinees participating in the study, 
their P&P score distributions were clearly different from those of the 
population, and it was determined that more sophisticated methods needed to 
be employed. (Author/SLD) 
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TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Abstract 



The purpose of this paper is to describe the techniques used in establishing the 
concordance tables between the TOEFL P&P and CBT section and total reported score 
scales. The TOEFL CBT consists of three sections Listening, Structure, and Reading 
just as in the P&P, plus a mandatory Essay session. Listening, Reading, and composite 
Structure and Essay scores plus a total score are reported for the CBT on scales that 
are non-overlapping with the P&P scales. Because of the new composite Structure and 
Essay score, it was unlikely that the Structure section and, hence, the total test could 
be equated to the P&P test, and hence a concordance study was conducted. 

In the concordance study, examinees took the CBT soon after their P&P 
administrations. Using the responses on both the P&P and the CBT forms of the study 
group, we estimated the conditional distributions of the CBT section scores and the 
observed Essay scores, given the P&P scores. Using this distribution and the marginal 
distributions of the P&P section scores estimated from the national population, we then 
obtained projected population distributions for the CBT scores and observed Essay 
scores. Given reference forms for the CBT test, we projected the CBT and Essay score 
distributions to reference-test observed score distributions which then served as the 
basis for concordance functions. 

Using the above techniques, we successfully established the concordance tables 
between the CBT and P&P reported score scales for TOEFL. One might argue that we 
could have used the study group alone to establish the concordance relationships. 
However, because of the self-selection of the examinees participated in the study, their 
P&P score distributions were clearly different from those of the population, and it was 
determined that more sophisticated methods needed to be employed. 
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Introduction 



After several years of development, TOEFL introduced its CBT version to over 
120 countries worldwide on July 24, 1998. The TOEFL CBT consists of three sections 
Listening (LC), Structure (ST), and Reading (RC) just as in the P&P, plus a mandatory 
Essay session. Both the Listening and the Structure sections are adaptive, but the 
Reading section is assembled linearly on the fly. Listening, Reading, and composite 
Structure and Essay scores plus a total score are reported for the CBT on scales that 
are non-overlapping with the P&P scales. Before the rollout of the TOEFL CBT, the 
Listening and the Structure reported scores ranged from 20 to 68, the Reading from 20 
to 67, and the total from 200 to 677. The CBT scales are from 0 to 30 for the sections 
and from 0 to 300 for the total reported scores, and the P&P section and total reported 
score scales are now truncated at 31 and 310, respectively. 

The CBT and the P&P tests of TOEFL are different not only in the ways they are 
given, but also in the measures themselves. There are many new item types for the CBT 
in the Listening and the Reading sections. For the Structure section, the reported score 
is now the composite of the Structure and the Essay scores. 

Because of the differences in the measures, and in particular because of the new 
composite Structure and Essay score, it was unlikely that the sections and, hence, the 
total test could be equated to the P&P test, and hence a concordance study was 
conducted. 
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The Data 



A total of 9,381 examinees from Nov 97, Dec 97, Jan 98, and Feb 98 TOEFL P&P 
administrations participated in the concordance study. They took the CBT soon after 
taking their P&P tests. After deleting records where the responses to any one of the 
sections were lacking or records that were duplicates, 9,247 of them remained. After 
matching these records with their P&P records, a total of 860 examinees were further 
removed because of the reasons listed below: 

• Their records were not used in P&P calibration: 

• Their CBT and P&P records could not be matched: 

• Their P&P records contained no valid reported scores. 



Motivation study 



For the remaining 8,387 examinees, their CBT scores (section estimated 0s) were 
compared with their P&P scores (section reported scores) to flag those who did 
exceptionally poor or well on the CBT. However, remember that the CBT section 9s and 
the P&P reported scores are not directly comparable. A remedy of this problem is to use 
the percentile ranks of the scores (the rank of the CBT 0 and that of the P&P reported 
score) because they are on the same scale and have the same Uniform distribution. 
Thus, the flagging of examinees who did exceptionally poor (dubbed under-motivated) or 
well (over-motivated) on the CBT was based on the following criteria: 

I^CBT — -^P&P | — Cr 0I " 

|bcBT - -kp&P | — C L 

where R CBT and R P&P are the percentile ranks of the CBT 0 and the P&P reported score 



of an examinee; L = In- — — is the logit of the percentile rank R and has a Normal 



distribution. While the constant c L here was set at twice the S.D. of L CBT - L P&P , c R was 
set at 0.15 (more or less arbitrarily) to exclude people whose percentile rankings on the 



4 



CBT and the P&P were too different (especially at the two extreme ends of the ranking 
scale). 



Figure 1 below shows the plot of the logits of CBT score rankings vs. those of the 
P&P score rankings for the LC measure. In Figure 1, the x-axis labeled TOEFL is the 
logit of P&P reported score ranking, and the y-axis labeled CONCORD is the logit of CBT 
0 ranking. The points outside the region bounded by the two lines are those flagged as 
having inconsistent performances on the CBT and the P&P LC measures. The patterns 
here are mainly caused by the discrete nature of the P&P reported score distribution. 

Insert Figure 1 about here 

Records of the 834 examinees who were flagged for any of the sections were 
deleted, leaving 7,553 records with valid and consistent scores on all the sections of the 
CBT as well as the P&P tests. These records were then matched with their CBT Essay 
records, and 496 of them could not be matched leaving us 7,057 complete records with 
P&P section responses, CBT section responses, and CBT Essay scores. 

A sample (the national sample) of 50,000 examinees representing the national 
population was assembled using the examinees from the same four P&P 
administrations. This sample had the above 7,057 examinees as a subset. Each 
examinee in the national sample had only P&P section responses. 

The Analysis 



Step 1 



Remember that we have item parameter estimates for the four P&P 
administrations and parameter estimates for the CBT items (they are all on the same 
P&P scale). Using the MGROUP estimation programs and assuming that the latent 
variables are Multivariate Normal, we got the estimated 0 P&P distribution for the 
national sample using their P&P section responses. MGROUP is a set of computer 
programs, each of which gives maximum likelihood estimates of y and E hi the 
regression model: 

0 = X T Y + e 

where e is Multivariate Normal with Mean O and Covariance E, 0 is a multidimensional 
latent variable and X is a user-specified design matrix. Each examinee’s vector of 
conditioning variables can serve as a row in the design matrix. Numerical 
approximation of a solution based on the E-M algorithm is used (Mislevy, 1985). 
Imputed ability estimates can be output, which are not “examinee optimal”, but 
designed to provide consistent estimates of subgroup distributions. 

Again using MGROUP and this time conditioning on the CBT Essay scores, we 
got the estimated (0 p&p ,0 CErr | Essay) distribution for the sample of 7,057 examinees 
with P&P, CBT, and Essay records. Among these 7,057 examinees, 501 of them with 
Essay score 0 were removed because we could not tell if their Essays were off-topic or 
blank due to a software problem 1 . For the 6,556 examinees left (the concordance 
sample) , we had the (0 P&P ,0 CBX | Essay) distribution. Along with the Essay score 
distribution, we got the joint distribution of (0 Pfi[P ,0 CBT , Essay) for the concordance 

sample. 

1 For some examinees who chose to key enter their Essays, their files might not be 
saved when their allotted time expired. This problem has since been corrected. 
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Step 2 



For the concordance sample, if the distribution of (0 P6[ p,9 C BT) is Multivariate 



Normal with Mean p = (Pp&p-Pcbt) 7 and Covariance 2 = 
distribution of (0 C bt I 0p&p = x p&p) is also Multivariate 



'N 



"p&p 

2 

CBT. P&P 

Normal 



'CBT.PStP 

^CBT 

with 



, the 
Mean 



PcBT + ^ CBT. P&P ^ P&P 



^p&p Pp&p ) and Covariance E CBT ^'cbt.p&p^'p&p^'cbt.p&p 



To show that the distribution of (0 PSl p,0 CBT )is indeed Multivariate Normal, we 
can show that the distributions of6 PS[PLC , (6 p &ps&we I^p&plc)> ^p&prc I ^p&p lc > ^p&p s&we ) * 
(^cbtlc l®p&p)> ^cbtst I ® p&p > ^ cbtlc ) > and (6 CBTRC | 9 p&p . 6 CBTLC , 6 CBTST ) are successively 
Normal. 



To show that the distribution of (6 CBTST | 0 PS[P ,6 CBXLC ) is Normal, for example, we 
only need to show that the residuals of the linear regression of 6 CBTST on 0 P&P and 
^cbtlc have a Normal distribution. This can be shown by the QQ Plot of the residuals 
(Plot of the quantiles of the residuals against those of the Standard Normal 
distribution) . 



Figure 2 below gives the QQ Plot of the residuals of the linear regression of 
6 cbxst on 0 P&P and 6 CBTLC . The plot is very close to being a straight line, indicating that 
the distribution of the residuals resembles closely that of a Normal distribution. 

Insert Figure 2 about here 



Now that we have shown the distribution of (0 P&P ,0 CBT ) for the concordance 
sample is Multivariate Normal, its Mean and Covariance can be estimated by the 
sample Mean m = (m p&p ,m CBT ) T = ((0.4574,0.4063,0.4343), (0.4763, 0.3406, 0.5577)) T and 



the sample Covariance 



S = 



S T 

° CBT. P&P 



’CBT.P&P 



'( 0.5887 0.4697 0.4535" 
0.4697 0.7668 0.6302 
lo.4535 0.6302 0.6379 
[0.5977 0.5008 0.5020^ 
0.4455 0.7109 0.5894 
0.4055 0.5780 0.5837 



/ 






0.5977 0.4455 0.4055 
0.5008 0.7109 0.5780 
0.5020 0.5894 0.5837 
0.6607 0.4867 0.4752 
0.4867 0.6677 0.5559 
0.4752 0.5559 0.5787 






Y\ 



< 



J) 



Since for the concordance sample (0p &p ,0 CBT )has a Multivariate Normal 



distribution, for any given x 



p&p ■ 



(0 



CBT 



0 PS[ p =x p& p)is also Multivariate Normal with 



Mean 



- 0.0035 + 0.9126XPSPLC -0.1035x p&pst + 0.2405x PS[PRC 
-0.0445 + 0.027 lXp^^ +0.8875x PS[ p ST + 0.0280x patPRC 
0. 1662 - 0.0368x PS[ p LC + 0.0149x P8cPSX +0.9264x 



P&PRC 



and Covariance 



0.0464 0.0121 0.0246") 
0.0121 0.0083 0.0156 
0.0246 0.0156 0.0442 
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The distribution of (Essay | 0 p&p ,0 CBr ) for the concordance sample can be 
estimated using logistic regression. Specifically, the logit of P(Essay > e) is linearly 
regressed on 0 P&P and 0 CBT for e=l,2,...,5, consecutively using the concordance sample 
data. The results are given below: 

logit(P(Essay > 1)) = 2.6455 -4.3 540# P&PLC + 46.76140 PSiPCT -17.79880 PSiPRC 

+ 7.38260 CBTL c — 49.2849# cbtst + 17.3622# cbtrc 

logit(P(Essay > 2)) = 2.7682 - 0.3298(9 PS!PLC - 3.25570 PSiPCT - O.4O540 PSiPRC 

+ 0.8197# CBTLC + 4.5897# cbtst + 0. 13960 cbtrc 

logit(P(Essay > 3)) = 0.5790 + 1.1 004# P&PLC - 4.64850 PSiPCT + O.966O0 PSiPRC 

- 0.6526# cbtlc + 6. 13620 CBrsx — 1 .2 172# CBTRC 



logit(P(Essay > 4)) = -3.4033 - 1.672 10 P&PLC + 1 1 . 18880 PSlP£n . - 3. 1O8O0 PSiPRC 

+ 3.O8290 CBTLC -1O.7OO20 cbtct + 2.218O0 CBTRC 

logit(P(Essay > 5)) = -8.4057 - 4.367 10 PS!PLC + 26.51820 p&P£n . - 6.27440 PSiPRC 

+ 7.066 1$c B tlc — 26.64 17# CBTST + 4.3429# CBTRC 



After having estimated the distributions of (0 CBT | 0 P&P )and (Essay | 0 PSlP ,0 CBT )for 
the concordance sample, we got the distribution of (0 CBT , Essay | 0 PSlP ). Assuming this 
distribution is the same for both the concordance and the national samples, we can get 
the joint distribution of (0 P&P ,0 CBT , Essay) for the national sample by using the 
distributions of (0 CBT , Essay | 0 P&P ) and 0 P&P for the national sample. For example, for a 
given point x p&p = (0.4340,0.3599,0. 0393) T from the 0 PS[P distribution for the national 



sample, we drew a value x CBT = (0.5296,0. 2829,0. 2502) r from the distribution of 

(®CBT I ®F 



= X, 



and Covariance 



^0.0464 

0.0121 



For (x P 8 !P ,x CBT ) , a value 3 for the Essay 



>) which is Multivariate Normal with Mean (0.3647,0.2878,0. 1920) T 
0.0121 0.0246^ 

0.0083 0.0156 
0.0246 0.0156 0.0442 
score was then drawn using the distribution of (Essay | 0 PSlP ,0 CBT ) which is given by 
P(Essay < e) = 0.00001, 0.03898, 0.37582, 0.74340, 0.97615, 1.00000 for e = 1, 2, ..., 
6 . 



Step 3 



Reference forms for the CBT sections were assembled linearly. These forms used 
the same types of items and conformed to the same sets of content specifications used 
in actual CBT tests (adjusted for test length if necessary). The CBT ST section uses the 
same reference form as used in the P&P test that consists of 38 items because the same 
item types are used in both the P&P and the CBT tests. The reference forms for the CBT 
LC and the RC sections consist of 50 and 44 items (compared to 50 and 49 items for 
the P&P LC and the RC sections),, respectively and have similar psychometric 
characteristics of the ones for the corresponding P&P sections. 
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For given values in the (0 P&P ,0 CBT ) space, using the P&P and the CBT reference 
form item parameters we can simulate the observed item response vectors to the P&P 
and the CBT tests (and thus the observed P&P and CBT section scores). Using 
resampling (sampling with replacement) from the joint distribution of 
(0 P&P ,0 CBT , Essay) for the national sample, we can then estimate their observed P&P and 
CBT section score and Essay score distributions. 

Step 4 



Recall that for the ST section in the CBT a composite score incorporating 
examinee performances on the Structure and on Essay is reported. In deriving the CBT 
ST composite score, we decided to give equal weight to Essay and the ST observed 
scores (adjusted for scale differences in Essay and the ST observed scores) using the 
following formula: 

CCBTST = X CBTST + jfg^J ‘ 

here X CBTST is the ST observed score, d(X CBTSr )and d(Essay)are the S.D. estimates of 
the CBT ST observed and Essay scores, respectively. From the distributions of the CBT 
ST observed scores and Essay scores, we have d(X CBTST ) = 6.4990 and 
d (Essay) = 0.9495 . 

Once we have determined how to compute the CBT ST composite score, its 
distribution can be estimated again using resampling techniques as in Step 3. This 
distribution along with the distributions of the P&P section observed and the CBT LC 
and the RC section observed scores then served as the basis for the concordance 
functions. 

The CBT total reported score is computed as ten-thirds of the sum of the LC, ST 
composite and RC reported scores just as in the P&P test. 

To convert the CBT observed section scores to the reported score scale, we need 
to first define the section reported score distribution as follows: 

• Standardize the observed LC, ST composite, and RC scores to the range from 0 to 
30: 

• Get total scores as ten-thirds of the sum of these standardized scores: 

• Obtain the distribution of the total scores and smooth it using a Negative 
Hypergeometric distribution: 

• Scale back this distribution to the range from 0 to 30. 

Figure 3 below shows the reported score distribution defined in the above 
manner. As is clear from Figure 3, this distribution is skewed to the left, which is 
commonly seen in distributions of test scores. It combines the characteristics of CBT 
observed section score distributions in a direct manner, and resembles the shape of the 
P&P total reported score distribution. The section reported scores with the above 
distribution have a Mean of 19.7 and S.D. of 4.9, and the total reported scores have a 
Mean of 195.1 and S.D. of 44.5. 

Insert Figure 3 about here 
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Step 5 



The conversion curves between CBT section observed and reported scores can be 
obtained by equipercentile equating using the CBT section reported score distribution 
obtained in Step 4, and the distributions of the CBT LC observed, ST composite, and RC 
observed scores. 

The P&P section observed scores are converted into reported scores using the 
conversion curves for the reference forms. 

Once we have obtained the section reported scores for the P&P and the CBT 
tests, we can get the frequencies of the P&P and the CBT total reported scores. Along 
with the frequencies of the P&P and the CBT section observed scores, we get a set of 
four concordance tables using equipercentile equating: 

P&P LC observed to CBT LC observed, 

P&P S&WE observed to CBT ST composite, 

P&P RC observed to CBT RC observed, and 
P&P total reported to CBT total reported. 

Figure 4 gives the plots of the concordance relationships between the P&P and 
the CBT tests for section observed and total reported scores. Except for the curve 
between the P&P ST observed scores and the CBT ST composite scores, these 
relationships have only minor curvatures throughout their range. The sudden change of 
curvature in the upper end of the concordance relationship between the P&P ST 
observed and the CBT ST composite scores is caused by the differences in the 
distributions of the two scores. Figure 5 gives the distributions of these two scores. The 
shape of the P&P ST observed score distribution is commonly seen. However, the 
distribution of the CBT ST composite scores exhibits unusual patterns caused by the 
combination of two totally different distributions. The Essay score distribution is more 
or less symmetric with most of the masses concentrating in the middle (at 3 or 4 point) 
while the CBT ST observed score distribution is similar in shape to that of the P&P ST 
observed scores. The combination of these two different distributions thus produces 
multi-modes in the distribution of the CBT ST composite scores as seen in Figure 5. 

Insert Figures 4 and 5 about here 

Using the conversion curves for the P&P sections, the concordance tables 
between the P&P and the CBT section observed scores, and the conversion curves for 
the CBT sections, we can get concordance tables between the P&P and the CBT section 
reported scores. Specifically, for a given P&P section reported score S P8tP , using the 
conversion curve for the P&P section and interpolation, we get the corresponding P&P 
observed section score X p&p . Using the concordance table and interpolation, we get an 

equivalent CBT observed section score X CBT , which is then converted into the reported 
score S CBT : 

S P8cP — > X P&P X CBT — ¥ S CBT 

Figure 6 below shows the concordance curves between the P&P and the CBT 
section reported scores. The patterns seen in the concordance between the P&P ST 
observed and the CBT ST composite scores carry over here. The concordance curve for 
the LC reported scores has change of steepness in the lower end, which is caused by 
the linear portion (below chance level) of the conversion curve for the P&P observed 
scores. 




) 
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Insert Figure 6 about here 

Summary 

Using the techniques described above, we successfully established the 
concordance tables between the CBT and P&P reported score scales for TOEFL which 
are shown in Tables 1 to 4 below. 

Insert Tables 1 to 4 about here 

One might argue that we could have used the study group alone to establish the 
concordance relationships. However, because of the self-selection of the examinees 
participated in the study, their P&P score distributions were clearly different from those 
of the population, and it was determined that something more sophisticated needed to 
be employed. Table 5 gives the Means of the P&P section observed scores and total 
reported scores for the concordance sample as well as for the national sample. 

Insert Table 5 about here 

As can be seen from table 5, because of self-selection, examinees who actually 
participated tended to be more able than those in the population. Thus complex 
mechanisms like those in the previous section are needed to reduce this artificial effect. 
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Figure 1. Plot of the logits of CBT score rankings 
vs . those of the P&P score rankings 
The LC measure . 
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CBT ST 



Figure 2. QQ Plot of residuals of CBT ST on P&P and CBT LC 
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Figure 3. CBT section reported score distribution. 
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Figure 4 . Plots of concordance curves for section observed 

and total reported scores 
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Figure 4 . Plots of concordance curves for section observed 
and total reported scores (cont.) 



Total reported 




BEST COPY AVAILABLE 



15 



o 

ERIC 






Figure 5 . Distributions of the P&P ST observed scores 
and the CBT ST composite scores 

P&P ST observed 




CBT ST composite 
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Figure 6. Plots of concordance curves for section reported scores 
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Table 1. Concordance between section reported scores 
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Table 2. Concordance between section reported scores 
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Table 3. Concordance between section reported scores 
Reading 
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33 


6 


62 


26 


52 


19 


42 


11 


32 


6 


61 


26 


51 


18 


41 


11 


31 


5 


60 


25 


50 


17 


40 


10 






59 


25 


49 


16 


39 


9 
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Table 4. Concordance between total reported scores 



p&p 


CBT 


P&P 


CBT 


P&P 


CBT 


P&P 


CBT 


677 


300 


583 


237 


490 


163 


397 


93 


673 


297 


580 


237 


487 


163 


393 


90 


670 


293 


577 


233 


483 


160 


390 


90 


667 


290 


573 


230 


480 


157 


387 


87 


663 


287 


570 


230 


477 


153 


383 


83 


660 ' 


287 


567 


227 


473 


150 


380 


83 


657 


283 


563 


223 


470 


150 - 


377 


80 


653 


280 


560 


220 


467 


147 


373 


77 


650 


280 


557 


220 


463 


143 


370 


77 


647 


277 


553 


217 


460 


140 


367 


73 


643 


273 


550 


213 


457 


137 


363 


73 


640 


273 


547 


210 


453 


133 


360 


70 


637 


270 


543 


207 


450 


133 


357 


70 


633 


267 


540 


207 


447 


130 


353 


67 


630 


267 


537 


203 


443 


127 


350 


63 


627 


263 


533 


200 


440 


123 


347 


63 


623 


263 


530 


197 


437 


123 


343 


60 


620 


260 


527 


197 


433 


120 


340 


60 


617 


260 


523 


193 


430 


117 


337 


57 


613 


257 


520 


190 


427 


113 


333 


57 


610 


253 


517 


187 


423 


113 


330 


53 


607 


253 


513 


183 


420 


110 


327 


50 


603 


250 


510 


180 


417 


107 


323 


50 


600 


250 


507 


180 


413 


103 


320 


47 


597 


247 


503 


177 


410 


103 


317 


47 


593 


243 


500 


173 


407 


100 


313 


43 


590 


243 


497 


170 


403 


97 


310 


40 


587 


240 


493 


167 


400 


97 






Table 5. Means of the P&P section observed scores and total reported 


scores 


Sample 




P&P LC 


P&P ST 


P&P RC 




Total 


National 




35.6 


26.6 


33.5 




528.7 


Concord 




36.6 


28.0 


35.9 




541.3 
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